Suspicious person detection device, suspicious person detection method, and program

ABSTRACT

The present invention provides a technology that can reduce erroneous detection and detect a suspicious person from an image at high accuracy. A suspicious person detection device according to one example embodiment of the present invention includes: an eye direction detection unit that detects an eye direction of a subject; a face direction detection unit that detects a face direction of the subject; an environment information acquisition unit that acquires environment information indicating arrangement of an object around the subject; and a determination unit that, based on the face direction, the eye direction, and the environment information, determines whether or not the subject is showing suspicious behavior.

This Application is a National Stage of International Application No.PCT/JP2017/024289 filed Jul. 3, 2017, claiming priority based onJapanese Patent Application No. 2016-133164 filed Jul. 5, 2016, thedisclosure of which is incorporated herein in its entirety by reference.

TECHNICAL FIELD

The present invention relates to a device, a method, and a program thatdetect a suspicious person from an image.

BACKGROUND ART

In a retail industry or a logistics industry, there is a problem of lossof items because of wrongdoing such as theft made by a customer or anemployee, that is, a problem of occurrence of missing-loss.Conventionally, it may be possible to install a surveillance cameraaround a store shelf or the like and inspect wrongdoing later byreviewing a captured image. However, inspection after an incident mayinvolve significant labor and danger. It is therefore desirable todetect a customer or an employee (suspicious person) who is showingsuspicious behavior before occurrence of an incident and takeprecautions such as questioning a suspicious person.

Advancement of information processing technologies and increase inresolution of surveillance cameras have made it possible to detect aperson from an image captured by a surveillance camera and, further,estimate the orientation of the face or the direction of a line of sightof the detected person. As application of such a technology, atechnology for automatically detecting a suspicious person from an imagehas been developed.

The technology disclosed in Patent Literature 1 is to acquire a facedirection and an eye direction of a person from an image and then detecta suspicious person based on a dispersion value or a motion amount ofthe line of sight or based on behavior that causes a large angledifference between the face direction and the eye direction.

CITATION LIST Patent Literature

-   PTL 1: Japanese Patent Application Laid-Open No. 2007-6427

SUMMARY OF INVENTION Technical Problem

In the technology disclosed in Patent Literature 1, however, asuspicious person is detected based only on a face direction and an eyedirection of a person to be detected. Thus, it is not possible toconsider the environment around a person resulting in occurrence oferroneous detection in which a person who is showing normal behaviorwith respect to a relationship with the environment is determined to bea suspicious person or, contrarily, a person who is showing abnormalbehavior is determined to be not a suspicious person.

The present invention has been made in view of the problems describedabove and intends to provide a technology that can reduce erroneousdetection and detect a suspicious person from an image at high accuracy.

Solution to Problem

A first example aspect of the present invention is a suspicious persondetection device including: an eye direction detection unit that detectsan eye direction of a subject; a face direction detection unit thatdetects a face direction of the subject; an environment informationacquisition unit that acquires environment information indicatingarrangement of an object around the subject; and a determination unitthat, based on the face direction, the eye direction, and theenvironment information, determines whether or not the subject isshowing suspicious behavior.

A second example aspect of the present invention is a suspicious persondetection method including steps of: detecting an eye direction of asubject; detecting a face direction of the subject; acquiringenvironment information indicating arrangement of an object around thesubject; and based on the face direction, the eye direction, and theenvironment information, determining whether or not the subject isshowing suspicious behavior.

A third example aspect of the present invention is a program that causesa computer to execute steps of: detecting an eye direction of a subject;detecting a face direction of the subject; acquiring environmentinformation indicating arrangement of an object around the subject; andbased on the face direction, the eye direction, and the environmentinformation, determining whether or not the subject is showingsuspicious behavior.

Advantageous Effects of Invention

According to the present invention, since suspicious behavior isdetermined based on environment information indicating arrangement of anobject around a subject in addition to the eye direction and the facedirection of the subject, erroneous detection can be reduced, and asuspicious person can be detected at high accuracy.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of a suspicious person detection systemaccording to a first example embodiment.

FIG. 2 is a general configuration diagram of the suspicious persondetection system according to the first example embodiment.

FIG. 3A is a schematic diagram illustrating an environment around asubject to be detected.

FIG. 3B is a schematic diagram illustrating an environment around asubject to be detected.

FIG. 4 is a schematic diagram of a face of a subject to be detected.

FIG. 5A is a schematic diagram of a face of a subject to be detected.

FIG. 5B is a schematic diagram of a face of a subject to be detected.

FIG. 5C is a schematic diagram of a face of a subject to be detected.

FIG. 6A is a schematic diagram of a face of a subject to be detected.

FIG. 6B is a schematic diagram of a face of a subject to be detected.

FIG. 7 is a block diagram of a suspicious person detection deviceaccording to the first example embodiment.

FIG. 8 is a schematic diagram of an exemplary detection result accordingto the first example embodiment.

FIG. 9A is a front view of a display displaying a notification of adetection result according to the first example embodiment.

FIG. 9B is a front view of a display displaying a notification of adetection result according to the first example embodiment.

FIG. 10 is a front view of a mobile terminal displaying a notificationof a detection result according to the first example embodiment.

FIG. 11 is a front view of a display displaying a searched detectionresult according to the first example embodiment.

FIG. 12 is a diagram illustrating a flowchart of a suspicious persondetection method according to the first example embodiment.

FIG. 13 is a diagram illustrating a flowchart of a suspicious persondetermination process according to the first example embodiment.

FIG. 14A is a schematic diagram illustrating a motion direction of asubject to be detected.

FIG. 14B is a schematic diagram illustrating a motion direction of asubject to be detected.

FIG. 15 is a block diagram of a suspicious person detection deviceaccording to a second example embodiment.

FIG. 16 is a general configuration diagram of a suspicious persondetection device according to each example embodiment.

DESCRIPTION OF EMBODIMENTS

While example embodiments of the present invention will be describedbelow with reference to the drawings, the present invention is notlimited to these example embodiments. Note that, throughout the drawingsdescribed below, components having the same function are labeled withthe same references, and the repeated description thereof may beomitted.

First Example Embodiment

FIG. 1 is a schematic diagram of a suspicious person detection system100 according to the present example embodiment. The suspicious persondetection system 100 has a suspicious person detection device 110, amobile terminal 130, and a surveillance camera recording device 150. Thesuspicious person detection device 110 and the surveillance camerarecording device 150 are connected to each other by wired connection orwireless connection via a network 190 such as the Internet. Thesuspicious person detection device 110 and the mobile terminal 130 areconnected by wireless connection such as wireless Local Area Network(LAN), a mobile communication, or the like. The surveillance camerarecording device 150 is a device that records images from one or morecameras 151. The suspicious person detection device 110 is a device thatdetects a suspicious person from images recorded by the surveillancecamera recording device 150 and displays a detection result on a display111. The mobile terminal 130 is a portable communication terminal thatdisplays a notification from the suspicious person detection device 110.The suspicious person detection system 100 may include other devicessuch as a server, a terminal, or the like.

FIG. 2 is a general configuration diagram of the suspicious persondetection system 100 according to the present example embodiment. Thesuspicious person detection device 110, the mobile terminal 130, and thesurveillance camera recording device 150 may be formed of a singledevice, respectively, or may be formed of two or more physicallyseparate devices connected via a wire or wirelessly. For example, thesuspicious person detection device 110 and the surveillance camerarecording device 150 may be a laptop computer, a desktop computer, aworkstation, a personal digital assistant, a server, a blade server, amainframe, an embedded system, or the like. For example, the mobileterminal 130 may be any communication terminal such as a personaldigital assistant, a mobile phone, a smartphone, or the like. Thespecific hardware configuration of the suspicious person detectiondevice 110, the mobile terminal 130, and the surveillance camerarecording device 150 is not limited to the following configuration butmay be of various types or forms.

The suspicious person detection device 110 and the surveillance camerarecording device 150 have configurations common to each other. Each ofthe suspicious person detection device 110 and the surveillance camerarecording device 150 has a processor 112, a memory 113, and a storagedevice 114. Further, each of the suspicious person detection device 110and the surveillance camera recording device 150 has a high-speedcontroller 115 including a high-speed interface and a low-speedcontroller 117 including a low-speed interface. The memory 113 and ahigh-speed expansion port 116 are connected to the high-speed controller115. Further, a display device such as the display 111 or the like isconnected to the high-speed controller 115. On the other hand, alow-speed expansion port 118 and the storage device 114 are connected tothe low-speed controller 117.

The processor 112, the memory 113, the storage device 114, thehigh-speed controller 115, the low-speed controller 117, and thehigh-speed expansion port 116 are connected to each other throughvarious buses. Further, the processor 112, the memory 113, the storagedevice 114, the high-speed controller 115, the low-speed controller 117,and the high-speed expansion port 116 may be implemented on a commonmotherboard or may be implemented in other forms as appropriate.

The processor 112 is a central processing unit (CPU), for example, andis able to process instructions executed within the suspicious persondetection device 110 and the surveillance camera recording device 150.Such instructions include an instruction that is used for displayinggraphics information of a graphical user interface (GUI) on a displaydevice such as the display 111 and stored in the memory 113 or thestorage device 114.

Further, a plurality of processors, a plurality of busses, or aplurality of processors and a plurality of busses can be used asappropriate together with a plurality of memory devices and multipletypes of memory devices. Further, a plurality of suspicious persondetection devices 110 or surveillance camera recording devices 150 canbe connected to each device that performs a part of the necessaryprocess. For example, a plurality of suspicious person detection devices110 or a plurality of surveillance camera recording devices 150 can beconnected to each other as a server bank, a group of blade servers, or amultiprocessor system.

The memory 113 stores therein information within the suspicious persondetection device 110 and the surveillance camera recording device 150.For example, the memory 113 may be a volatile memory unit, anon-volatile memory unit, or a combination thereof. The memory 113 maybe another computer readable storage medium, such as a magnetic disk, anoptical disk, or the like, for example.

The storage device 114 can configure mass storage used for thesuspicious person detection device 110 and the surveillance camerarecording device 150. The storage device 114 may be, for example, acomputer readable storage medium such as a floppy (registered trademark)disk device, a hard disk device, an optical disk device, a tape device,a solid state memory device such as a flash memory, a disk array, or thelike or may include such a computer readable storage medium. The storagedevice 114 may be a storage area network or a device with anotherconfiguration.

The high-speed controller 115 manages processes in which the bandwidthfor the suspicious person detection device 110 and the surveillancecamera recording device 150 is intensively used. On the other hand, thelow-speed controller 117 manages processes in which the bandwidth isless intensively used. However, such allocation of the functions is amere example, and allocation is not limited thereto. Further, a part ora whole of the high-speed controller 115 may be incorporated in theprocessor 112.

The high-speed controller 115 is connected to the high-speed expansionport 116 that can accept the memory 113 and various expansion cards.Further, the high-speed controller 115 is connected to the display 111via a graphics processor or an accelerator, for example.

The low-speed controller 117 is connected to the storage device 114 andthe low-speed expansion port 118. The low-speed expansion port 118 caninclude, for example, a communication port of various standards such asUniversal Serial Bus (USB), Bluetooth (registered trademark), wired orwireless Ethernet (registered trademark), or the like. One or aplurality of input devices 119 such as a keyboard, a pointing device, ascanner, or the like can be connected to the low-speed expansion port118. Furthermore, one or a plurality of cameras 115 are connected to thelow-speed expansion port 118 in the surveillance camera recording device150 in particular. Further, one or a plurality of network devices suchas a switch, a router, or the like can be connected to the low-speedexpansion port 118 via a network adapter, for example. That is, thelow-speed expansion port 118 functions as a communication interface.

The suspicious person detection device 110 and the surveillance camerarecording device 150 can be implemented in many different forms withoutbeing limited to the forms described above. For example, the suspiciousperson detection device 110 and the surveillance camera recording device150 can be implemented in a form of a typical server or a plurality ofservers in a form of a group of such servers. Further, the suspiciousperson detection device 110 and the surveillance camera recording device150 can be implemented as a part of the rack server system. Furthermore,the suspicious person detection device 110 and the surveillance camerarecording device 150 can be implemented in a form of a personal computersuch as a laptop computer, a desktop computer, or the like. Further, thesuspicious person detection device 110 and the surveillance camerarecording device 150 may be combined to be implemented as a singledevice.

Note that a part or the whole of the program executed by the processor112 of the suspicious person detection device 110 and the surveillancecamera recording device 150 can be provided by a computer readablestorage medium storing the above, such as a digital versatile disc-readonly memory (DVD-ROM), a compact disc-read only memory (CD-ROM), a flashmemory such as a USB memory or the like.

The mobile terminal 130 has a processor 131, a memory 132, a display133, a communication interface 134, a camera 135, and an electroniccompass 136. The mobile terminal 130 may have a storage device such as amicro-drive, a flash memory, or the like as an additional storage.Respective components of the mobile terminal 130 are connected to eachother by using various buses. At least some of the components of themobile terminal 130 may be attached on a common motherboard or may beattached in a different manner.

The processor 131 can execute an instruction within the mobile terminal130, such as an instruction stored in the memory 132. The processor 131may be implemented as a chip or a chipset having one or a plurality ofanalog or digital processors. The processor 131 controls othercomponents of the mobile terminal 130 to perform display of a userinterface, execution of an application, wireless communication, or thelike, for example.

The processor 131 can input and output information to the user through adisplay interface 139 connected to the display 133 and the inputinterface 140. Any display unit such as a liquid crystal display, anorganic electroluminescence (EL) display, or the like, for example maybe used as the display 133. The display interface 139 has a circuitrequired for driving the display 133 so as to present an image and otherinformation to the user. Further, the display 133 is configured as atouchscreen and outputs a user operation as an electrical signal. Theinput interface 140 receives electrical signals indicating a useroperation performed on the display 133 and performs conversion of theseelectrical signals for transmission to the processor 131.

The memory 132 stores therein information present in the mobile terminal130. The memory 132 may be a computer readable storage medium, avolatile memory unit, a non-volatile memory unit, or a combinationthereof, for example. As the memory 132, for example, a flash memory, anon-volatile random access memory (NVRAM), or the like may be used.

The camera 135 is an image capturing unit that outputs an image of acapturing result as digital data. As the camera 135, any image capturingdevice such as a Charge Coupled Device (CCD) camera, a ComplementaryMetal Oxide Semiconductor (CMOS) camera, or the like may be used. Thecamera 135 includes an image pickup device, an electric circuit, a lens,or the like necessary for image capturing.

The electronic compass 136 is an acquisition unit that acquires theorientation of the mobile terminal 130. As the electronic compass 136,any device such as a geomagnetism sensor that is capable of acquiringthe orientation of the mobile terminal 130 may be used.

The communication interface 134 is connected to a global positioningsystem (GPS) receiver 137 and a mobile communication device 138. Themobile communication device 138 performs mobile communication using anymode or protocol including voice communication of a Global System forMobile communication (GSM) (registered trademark) scheme or the like,messaging of a Short Message Service (SMS) or the like, and datacommunication of a Code Division Multiple Access (CDMA) scheme or thelike. The mobile communication device 138 includes a processor, anelectric circuit, an antenna, or the like necessary for mobilecommunication.

The mobile terminal 130 can communicate in voice by using an audiointerface 141. The audio interface 141 is connected to a microphone anda speaker. The audio interface 141 receives utterance information from auser through a microphone and converts the utterance information intodigital information that can be processed by the processor 131. Further,the audio interface 141 can generate an audible sound to the userthrough a speaker, for example. The audio output by the audio interface141 includes an audio from voice communication, an audio recorded in thememory 132 (for example, a voice message, a music file, or the like), oran audio generated by an application operating on the mobile terminal130.

The mobile terminal 130 is provided with an extension interface 142 andan external interface 143. An extended memory is connected to theextension interface 142, for example. The extended memory may functionas additional storage to the mobile terminal 130 and may store a programrunning on the mobile terminal 130 or other information. Further, theextended memory may store secure information and, in such a case,function as a security module.

The external interface 143 is configured to communicate with theprocessor 131 and can perform wired communication or wirelesscommunication. The external interface 143 performs near fieldcommunication with Bluetooth (registered trademark) or the like orwireless communication with a wireless LAN or the like between themobile terminal 130 and another device, for example. As the externalinterface 143, a plurality of interfaces may be combined and used.

FIG. 3A and FIG. 3B are schematic diagrams illustrating an environmentaround a subject A in a suspicious person detection in the presentexample embodiment. FIG. 3A and FIG. 3B illustrate states where thesubject A and a store shelf C are viewed from the upper part in thegravity direction. In general, since an aisle is arranged beside thestore shelf C, a direction along the isle can be defined as an aisledirection D, and a direction toward the store shelf C can be defined asan item direction E. The aisle direction D and the item direction E aredefined in any way in accordance with the layout of the store shelf Cand the aisle.

The subject A has a field of view B that is a range visible along theline of sight direction. As normal behavior, when the subject A ismoving to the store shelf C that displays an intended item, the subjectA directs the field of view B to the aisle direction D as illustrated inFIG. 3A. As another normal behavior, when the subject A is watching thestore shelf C that displays an intended item, the subject A directs thefield of view B to the item direction E as illustrated in FIG. 3B. Thesuspicious person detection system 100 according to the present exampleembodiment determines the above behavior as normal behavior and detectsbehavior different from the above behavior as suspicious behavior.

While FIG. 3A and FIG. 3B illustrate the store shelf C used fordisplaying items in a retail industry, a belt conveyer or a cargo usedfor transporting items corresponds to the store shelf C in a logisticsindustry. In this case, the direction toward the belt conveyer or thecargo is the item direction E, and it may be possible that an employeesteals an item while watching the aisle direction D.

FIG. 4 is a schematic diagram of a face F of the subject A showingnormal behavior. Further, FIG. 5A to FIG. 5C and FIG. 6A to FIG. 6B areschematic diagrams of the face F of the subject A showing suspiciousbehavior. The orientation of the line of sight of the subject A isdefined as an eye direction G, and the orientation of the face F of thesubject A is defined as the face direction H. In FIG. 4, FIG. 5A to FIG.5C, and FIG. 6A to FIG. 6B, the eye direction G and the face direction Hare represented by arrows. As illustrated in FIG. 4, in normal behavior,the eye direction G and the face direction H are substantially matched.

As an example of suspicious behavior, FIG. 5A illustrates a state wherethe eye direction G and the face direction H are mismatched from eachother. Typically, when watching an item, the subject A often moves theface and directs the line of sight to the item. In contrast, when tryingto steal an item, the subject A often checks the surrounding situation.At this time, to avoid being suspected by others, the subject A may moveonly the line of sight while facing the item to check the surroundingsituation. Thus, determination whether or not there is a mismatch statewhere the eye direction G and the face direction H are significantlydifferent from each other as illustrated in FIG. 5A can be used fordetection of a suspicious person.

As another example of suspicious behavior, each of FIG. 5B and FIG. 6Aillustrates a state of significant and frequent motion of the facedirection H. The subject A is moving the face direction H in the lateraldirection in FIG. 5B, and the subject A is moving the face direction Hin the vertical direction in FIG. 6A. When trying to steal an item, thesubject A may move the face to look around and check the surrounding.Thus, determination whether or not there is a face-shaking state wherethe subject A moves the face significantly and frequently as illustratedin FIG. 5B and FIG. 6A can be used for detection of a suspicious person.

As another example of suspicious behavior, each of FIG. 5C and FIG. 6Billustrates a state of significant and frequent motion of the eyedirection G. The subject A is moving the eye direction G in the lateraldirection in FIG. 5C, and the subject A is moving the eye direction G inthe vertical direction in FIG. 6B. When trying to steal an item, thesubject A may move the line of sight to look around and check thesurrounding. Thus, determination whether or not there is an eye-movingstate where the subject A moves the line of sight significantly andfrequently as illustrated in FIG. 5C and FIG. 6B can be used fordetection of a suspicious person.

Note that not only a suspicious person necessarily shows the behaviorillustrated in FIG. 5A to FIG. 5C and FIG. 6A to FIG. 6B but also anormal customer or employee may show such behavior. Thus, the suspiciousperson detection system 100 according to the present example embodimentdetermines a suspicious person based on a predetermined criterion usingenvironment information on the environment around a subject as describedlater in addition to the behavior of the subject as illustrated in FIG.5A to FIG. 5C and FIG. 6A to FIG. 6B.

FIG. 7 is a block diagram of the suspicious person detection device 110according to the present example embodiment. In FIG. 7, lines betweenblocks indicate main flows of data, and there may be other data flowthan is illustrated in FIG. 7. In FIG. 7, each block illustrates aconfiguration in a unit of function rather than a configuration in aunit of hardware (device).

The suspicious person detection device 110 includes a detection resultstorage unit 1101, an image processing unit 1102, a face regionextraction unit 1103, an eye direction detection unit 1104, a facedirection detection unit 1105, an environment information acquisitionunit 1106, a suspicious person determination unit 1107, a resultnotification unit 1108, a result search unit 1109, and a result displayunit 1110. In the suspicious person detection device 110, the imageprocessing unit 1102, the face region extraction unit 1103, the eyedirection detection unit 1104, the face direction detection unit 1105,the environment information acquisition unit 1106, the suspicious persondetermination unit 1107, the result notification unit 1108, the resultsearch unit 1109, and the result display unit 1110 are stored in thememory 113 as programs executable by the processor 112, respectively.That is, when performing a suspicious person detection method accordingto the present example embodiment, the processor 112 functions as theimage processing unit 1102, the face region extraction unit 1103, theeye direction detection unit 1104, the face direction detection unit1105, the environment information acquisition unit 1106, the suspiciousperson determination unit 1107, the result notification unit 1108, theresult search unit 1109, and the result display unit 1110. At least someof these functions may be implemented as an electric circuit rather thana program. Further, the detection result storage unit 1101 is a part ofthe memory 113 or the storage device 114. That is, when performing thesuspicious person detection method according to the present exampleembodiment, the memory 113 or the storage device 114 functions as thedetection result storage unit 1101.

The image processing unit 1102 acquires an image that is a moving imagefile recorded in the surveillance camera recording device 150 anddivides it into images that are static image files. Specifically, theimage processing unit 1102 divides a moving image file on a frame basisand thereby generates images that are static image files. The imageprocessing unit 1102 may acquire a moving image file which is beingrecorded by the surveillance camera recording device 150 at any time ona predetermined time basis (for example, every second) or maycollectively acquire moving image files which have already been recordedby the surveillance camera recording device 150. Furthermore, the imageprocessing unit 1102 may perform any image processing such as noisereduction, brightness adjustment, contrast adjustment, or the like on animage. The image processing unit 1102 holds the acquired image in thememory 113 or the storage device 114.

Subsequent processes may be performed on images of all the frames or maybe performed on images of frames that are reduced on a predeterminednumber basis in accordance with the processing speed of a computer and arequired accuracy. Further, in the subsequent processes, an image thatis a moving image file may be used without generating images that isstatic image files. In such a case, subsequent face region extraction,eye direction detection, face direction detection, environmentinformation acquisition, and suspicious person determination areperformed with each frame in the moving image file being defined as oneimage.

The face region extraction unit 1103 extracts a region including a faceof a person by using any method from an image acquired from the imageprocessing unit 1102. For example, the face region extraction unit 1103acquires luminance of each pixel of the image acquired by the imageprocessing unit 1102 and determines, as a face region, a set of pixelsin which the luminance of neighboring pixels is greater than apredetermined threshold. The face region extraction unit 1103 holds theposition of the extracted face region (for example, coordinates of theleft end, the right end, the upper end, and the lower end of the faceregion) in the memory 113 or the storage device 114. Extraction of aface region is not limited to that described here but may be performedby any method that can extract a region including a face of a personfrom an image. When two or more face regions are extracted from oneimage, subsequent eye direction detection, face direction detection, andsuspicious person determination are performed for each of the faceregions.

The face region extraction unit 1103 provides, to each face region, atracking ID that is an identifier used for tracking a person displayedin the face region. In an image of a frame to be processed, when a faceregion is detected near the position of a face region extracted in animage of the previous frame, the face region extraction unit 1103provides the same tracking ID to the face region. When there is no faceregion corresponding to the image of the previous frame, the face regionextraction unit 1103 provides a new tracking ID to a face region of animage of a frame to be processed. Thereby, after a face of a person wasdetected and until the face of the person is no longer detected, a faceregion in which the same person is likely to be included is tracked in atemporal manner, and scores may be accumulated.

To generate a tracking ID, the face region extraction unit 1103 may useany face recognition technology to recognize a person displayed in aface region. That is, the face region extraction unit 1103 provides thesame tracking ID to a face region in which the same person identified byusing a face recognition technology is displayed. Thereby, tracking canbe continued even when a face of a person was no longer detected andthen a face of the same person is again detected.

The eye direction detection unit 1104 extracts an eye direction by usingany method from a face region extracted by the face region extractionunit 1103. For example, the eye direction detection unit 1104 acquiresthe positions of pupils of both eyes by using template matching from aface region extracted by the face region extraction unit 1103. The eyedirection detection unit 1104 then calculates the direction of the lineof sight for each of both the eyes based on the relative position of apupil to a predetermined reference point on a face of a person. Thepredetermined reference point may be a position of a Purkinje image thatis a light reflection point on a cornea, or an inner canthus, or thelike, for example. The eye direction detection unit 1104 calculates theeye direction by using the relative position of a pupil to apredetermined reference point in a face region based on a tableillustrating a pre-recorded relationship between the relative positionof a pupil to a predetermined reference point and the eye direction. Thetable indicating the relationship between a relative position of a pupiland an eye direction is determined through an experiment or a simulationin advance. The eye direction detection unit 1104 holds the detected eyedirection in the memory 113 or the storage device 114. Detection of theeye direction is not limited to that illustrated here but may beperformed by using any method that can detect the direction of the lineof sight of a subject from an image.

The face direction detection unit 1105 extracts a face direction byusing any method from a face region extracted by the face regionextraction unit 1103. For example, the face direction detection unit1105 acquires the positions of both eyes and the position of a nose byusing template matching from a face region extracted by the face regionextraction unit 1103. The face direction detection unit 1105 thendefines a line connecting the middle point of both the eyes to theposition of the nose as a center line of the face and calculates theratio of the distance from the center line to the left end of a faceregion and the distance from the center line to the right end of theface region. The face direction detection unit 1105 then calculates alateral face direction by using the ratio of the left and the right inthe face region based on a table indicating a pre-recorded relationshipbetween a lateral ratio and a face direction. The table indicating therelationship between a lateral ratio and a face direction is determinedthrough an experiment or a simulation in advance. Similarly, the facedirection detection unit 1105 uses the position of a component of a faceto calculate the face direction in the vertical direction. For example,a line connecting the positions of both eyes is defined as a horizontalline of the face, and the ratio of the distance from the horizontal lineto the upper end of a face region and the distance from the horizontalline to the lower end of the face region is calculated. The facedirection detection unit 1105 then calculates a vertical face directionby using the vertical ratio in a face region based on a table indicatinga pre-recorded relationship between a vertical ratio and a facedirection. The face direction detection unit 1105 holds the detectedface direction in the memory 113 or the storage device 114. Detection ofthe face direction is not limited to that illustrated here but may beperformed by using any method that can detect the orientation of a faceof a subject from an image.

The environment information acquisition unit 1106 acquires environmentinformation on an environment around a face region extracted by the faceregion extraction unit 1103. The environment information in the presentexample embodiment is information indicating arrangement of an objectaround a person included in a face region, that is, informationindicating a region or a position where an object other than a person islocated. For example, in a retail industry, the environment informationis information indicating a region or a position where the store shelf Cused for displaying items as illustrated in FIG. 3A and FIG. 3B islocated. Further, in a logistics industry, the environment informationis information indicating a region or a position where a belt conveyeror a cargo used for transporting items is located.

The environment information acquisition unit 1106 may automaticallydetect environment information from an image acquired by the imageprocessing unit 1102. Typically, an object arranged around a person doesnot move. Thus, the environment information acquisition unit 1106 mayacquire, as environment information, a region which has not moved for apredetermined time period or longer except a background in an imagecaptured by a single camera. The background can be identified as aregion having a predetermined color. Further, the environmentinformation acquisition unit 1106 may recognize an object other than aperson in an image by an object recognition technology using templatematching, or the like and acquire a region in which the object isdisplayed as environment information.

As another method, the environment information acquisition unit 1106 mayacquire, as environment information, a designated region recorded in thestorage device 114 in advance. In this case, a range of a region to beidentified as an object other than a person within an image captured bya single camera is designated by the input device 119 and recorded inthe storage device 114 in advance. The environment informationacquisition unit 1106 then reads and acquires, as environmentinformation, the range of the region to be identified as an objectwithin an image from the storage device 114.

As yet another method, the environment information acquisition unit 1106may acquire a map of object arrangement (that is, a set of positions)recorded in the storage device 114 in advance as environmentinformation. In this case, a map including at least object arrangementin a range captured by a camera is recorded in the storage device 114 inadvance. The environment information acquisition unit 1106 then readsand acquires a map indicating object arrangement as environmentinformation from the storage device 114. A method of acquiringenvironment information indicating arrangement of an object other than asubject is not limited to that illustrated here, and any method may beused.

The suspicious person determination unit 1107 performs determination ofa suspicious person based on the eye direction and the face directiondetected by the eye direction detection unit 1104 and the face directiondetection unit 1105 and the environment information acquired by theenvironment information acquisition unit 1106. The suspicious persondetermination unit 1107 performs three types of determination based onthe eye direction and the face direction of a face region to bedetermined and the environment information and accumulates scores basedon the determination. The score is a value representing the degree ofsuspiciousness of a face region to be determined, and the resultnotification unit 1108 changes the score in accordance with the resultof determination described below. In the present example embodiment, ahigher score is set for a higher degree of suspiciousness, and apredetermined value is added to the score when the determination resultfrom the result notification unit 1108 indicates suspicious behavior. Incontrast, a lower score may be set for a higher degree ofsuspiciousness, and a predetermined value may be subtracted from thescore when the determination result from the result notification unit1108 indicates suspicious behavior.

[Mismatch Determination]

The suspicious person determination unit 1107 determines that a faceregion to be determined is in the mismatch state when the differencebetween the eye direction G and the face direction H is larger than apredetermined threshold as illustrated in FIG. 5A and when the facedirection H is oriented to an object other than a person. Specifically,first, the suspicious person determination unit 1107 determines whetheror not the face direction H is oriented to at least one object indicatedby environment information acquired by the environment informationacquisition unit 1106 (that is, the position of an object). A statewhere a person faces an object can be determined by an angle at whichthe face direction H enters a region of the object indicated byenvironment information within an image. That is, an incident angle of aface direction to an object region is larger when a face of a personfully faces the object, and the incident angle is smaller when the faceof the person does not face the object at all. Thus, when an angle atwhich the face direction H enters a region of an object is larger thanor equal to a predetermined value, it is determined that the personfaces the object. The suspicious person determination unit 1107calculates an angle difference (absolute value) relative to theX-direction of a face region and an angle difference (absolute value)relative to the Y-direction as the difference between the eye directionG and the face direction H, respectively. In the present exampleembodiment, the X-direction is defined as the horizontal direction(lateral direction) of an image including a face region, and theY-direction is defined as the perpendicular direction (verticaldirection) of the image including the face region. Other tow directionswithin an image including a face region may be used without beinglimited to the X-direction and the Y-direction. Further, when an angledifference relative to the X-direction is larger than a threshold forthe X-direction or an angle difference relative to the Y-direction islarger than a threshold for the Y-direction and, furthermore, when theface direction H faces an object indicated by environment information,the suspicious person determination unit 1107 determines that a faceregion to be determined is in the mismatch state. To determine thedifference between the eye direction G and the face direction H, thesedirections may be directly compared without being divided into theX-direction and the Y-direction of a face region. Further, any methodthat can quantitively determine the difference between the eye directionG and the face direction H may be used.

When determining that a face region to be determined is in the mismatchstate, the suspicious person determination unit 1107 adds apredetermined addition score value to an accumulated score of thetracking ID provided to the face region. The accumulated score is usedas a condition to perform notification performed by the resultnotification unit 1108 described later.

The suspicious person determination unit 1107 may use different additionscore values for respective thresholds. For example, a first additionscore value is used when the difference between the eye direction G andthe face direction H is larger than a first threshold and smaller thanor equal to a second threshold, and a second addition score value isused when the difference between the eye direction G and the facedirection H is larger than the second threshold. With such aconfiguration, the accumulated score can be finely controlled inaccordance with the level of the difference between the eye direction Gand the face direction H, and notification can be properly performed.

As the addition score value and the threshold for the difference betweenthe eye direction G and the face direction H, values defined in advancethrough an experiment or a simulation may be used or values input by theinput device 119 may be used, respectively.

[Face-Shaking Determination]

The suspicious person determination unit 1107 determines that a faceregion to be determined is in the face-shaking state when the facedirection H is moved significantly and frequently to satisfy apredetermined reference as illustrated in FIG. 5B and FIG. 6A and,furthermore, when the center of motion of the face direction H isoriented to an object other than a person. Specifically, the suspiciousperson determination unit 1107 acquires the face direction H from apredetermined time to the time of capturing a face region to bedetermined. Next, the suspicious person determination unit 1107calculates the motion angle (absolute value) relative to the X-directionand the motion angle (absolute value) relative to the Y-direction of theface direction H in a temporal manner, respectively. In the presentexample embodiment, the X-direction is defined as the horizontaldirection (lateral direction) of an image including a face region, andthe Y-direction is defined as the perpendicular direction (verticaldirection) of the image including the face region. Other two directionswithin an image including a face region may be used without beinglimited to the X-direction and the Y-direction. Further, the suspiciousperson determination unit 1107 calculates the number of times thatcertain motion occurs from a predetermined time to the time of capturinga face region to be determined, and the certain motion is such that themotion angle relative to the X-direction is larger than a threshold forthe X-direction or the motion angle relative to the Y-direction islarger than a threshold for the Y-direction and the center of the motionis directed to at least one object indicated by environment informationacquired by the environment information acquisition unit 1106. Thesuspicious person determination unit 1107 determines that a face regionto be determined is in the face-shaking state when the calculated numberof times is greater than a predetermined threshold. To determine themotion of the face direction H, the motion distance of an end of a linerepresenting the face direction H on a face region may be used insteadof the angle of the face direction H. Further, any method that canquantitively determine motion of the face direction H may be used.

When determining that a face region to be determined is in theface-shaking state, the suspicious person determination unit 1107 adds apredetermined addition score value to an accumulated score of thetracking ID provided to the face region. As a threshold for the motionangle of the face direction H, a time period, the number of times, andan addition score value, values defined in advance through an experimentor a simulation may be used or values input by the input device 119 maybe used, respectively.

[Eye-Moving Determination]

The suspicious person determination unit 1107 determines that a faceregion to be determined is in the eye-moving state when the eyedirection G is moved significantly and frequently to satisfy apredetermined reference as illustrated in FIG. 5C and FIG. 6B and,furthermore, when the face direction H is oriented to an object otherthan a person. Specifically, first, the suspicious person determinationunit 1107 determines whether or not the face direction H is directed toat least one object indicated by environment information acquired by theenvironment information acquisition unit 1106. The suspicious persondetermination unit 1107 acquires the eye direction G from apredetermined time to the time of capturing a face region to bedetermined. Next, the suspicious person determination unit 1107calculates the motion angle (absolute value) relative to the X-directionand the motion angle (absolute value) relative to the Y-direction of theeye direction G in a temporal manner, respectively. In the presentexample embodiment, the X-direction is defined as the horizontaldirection (lateral direction) of an image including a face region, andthe Y-direction is defined as the perpendicular direction (verticaldirection) of the image including the face region. Other two directionswithin an image including a face region may be used without beinglimited to the X-direction and the Y-direction. Further, the suspiciousperson determination unit 1107 calculates the number of times thatcertain motion occurs from a predetermined time to the time of capturinga face region to be determined, and the certain motion is such that themotion angle for the X-direction is larger than a threshold for theX-direction or the motion angle for the Y-direction is larger than athreshold for the Y-direction in a state where the face direction H isoriented to at least one object indicated by environment informationacquired by the environment information acquisition unit 1106. Thesuspicious person determination unit 1107 determines that a face regionto be determined is in the eye-moving state when the calculated numberof times is greater than a predetermined threshold. To determine themotion of the eye direction G, the motion distance of an end of a linerepresenting the eye direction G on a face region may be used instead ofthe angle of the eye direction G. Further, any method that canquantitively determine motion of the eye direction G may be used.

When determining that a face region to be determined is in theeye-moving state, the suspicious person determination unit 1107 adds apredetermined addition score value to an accumulated score of thetracking ID provided to the face region. As a threshold for the motionangle of the eye direction G, a time period, the number of times, and anaddition score value, values defined in advance through an experiment ora simulation may be used or values input by the input device 119 may beused, respectively.

The suspicious person determination unit 1107 may perform the whole or apart of the mismatch determination, the face-shaking determination, andthe eye-moving determination. Further, the suspicious persondetermination unit 1107 may perform other determination that candetermine a suspicious person based on an eye direction, a facedirection, and environment information.

As the eye direction used for determination by the suspicious persondetermination unit 1107, an average value of respective eye directionsof both eyes may be used, or an eye direction of one of both eyes may beused. Alternatively, the suspicious person determination unit 1107 mayperform determination on both eyes, respectively.

As discussed above, the suspicious person detection device 110 accordingto the present example embodiment performs determination of suspiciousbehavior based on environment information indicating arrangement of oneor more objects around a subject in addition to the face direction andthe eye direction of the subject. This enables reduced erroneousdetection and more accurate determination of a suspicious personcompared to determination based on only the face direction and the eyedirection.

The suspicious person determination unit 1107 records the determinedstate and score as a detection result in a predetermined form in thedetection result storage unit 1101 together with information output fromthe face region extraction unit 1103, the eye direction detection unit1104, and the face direction detection unit 1105. FIG. 8 is a schematicdiagram of an exemplary detection result 1101 a recorded in thedetection result storage unit 1101. The detection result 1101 a may berecorded as one or a plurality of data files or may be recorded as oneor a plurality of databases. The detection result 1101 a includescapturing time of an image acquired by the image processing unit 1102, acamera ID used for identifying a camera that has captured an image, aface image obtained by cutting out a position at and near a face regionextracted by the face region extraction unit 1103, or a tracking ID usedfor tracking a subject of the face region. The position of a face regionis represented by the X-coordinates of the left end and the right end ofa face and the Y-coordinates of the upper end and the lower end thereof.

Further, the detection result 1101 a includes eye positions and an eyedirection detected by the eye direction detection unit 1104 at the timeof interest and a face direction detected by the face directiondetection unit 1105. The eye position is represented by the X-coordinateand Y-coordinate of the pupil for each of both eyes. The eye directionis represented by two angles of an azimuth angle θ and an elevationangle ϕ for each of both eyes. Instead, an eye direction may berepresented by an average value of eye directions of both eyes. A facedirection is represented by three angles of a pan angle, a tilt angle,and a roll angle. A face position, eye positions, an eye direction, anda face direction may be represented by any scheme that can define theposition or the direction, respectively, without being limited to thoseillustrated here. Furthermore, the detection result 1101 a includes astate determined by the suspicious person determination unit 1107 and anaccumulated score at the time of interest. The form of the detectionresult 1101 a illustrated here is an example and may be recorded in anyform.

The result notification unit 1108 notifies a surveillant of a detectionresult determined by the suspicious person determination unit 1107 byusing at least one of the display 111 connected to the suspicious persondetection device 110 and the mobile terminal 130 communicating with thesuspicious person detection device 110. For example, a surveillant maybe a shop clerk when monitoring wrongdoing of a customer and may be amanager when monitoring wrongdoing of an employee. A specificnotification method will be described later by using FIG. 9A, FIG. 9B,and FIG. 10.

The result search unit 1109 receives a condition input by a surveillantfrom the input device 119 connected to the suspicious person detectiondevice 110 and searches the detection result storage unit 1101 toacquire a detection result which matches the condition. The resultdisplay unit 1110 displays the detection result searched for by theresult search unit 1109 on the display 111 connected to the suspiciousperson detection device 110. A specific display method of a detectionresult will be described later by using FIG. 11.

FIG. 9A and FIG. 9B are front views of the display 111 displaying anotification of a detection result from the result notification unit1108. The result notification unit 1108 displays the notification screen111 a on the display 111. The notification screen 111 a includes adetection result detected by the suspicious person determination unit1107 together with an image captured by the surveillance camerarecording device 150. The image included in the notification screen 111a may be any of a moving image and a static image.

As illustrated in FIG. 9A, the result notification unit 1108 displays asuperimposed first frame 111 c so as to surround a face region 111 bextracted by the face region extraction unit 1103 on an image beingdisplayed. Further, the result notification unit 1108 displays asuperimposed state indication 111 e near the face region 111 b extractedby the face region extraction unit 1103 on the image being displayed.The state indication 111 e includes a text representing a statedetermined by the suspicious person determination unit 1107 and anaccumulated score. When a face region corresponds to at least one of themismatch state, the face-shaking state, and the eye-moving state as astate included in the state indication 111 e, the result notificationunit 1108 displays a text representing the state, and when a face regioncorresponds to none of these states, the result notification unit 1108displays “normal”.

When a face region corresponds to at least one of the mismatch state,the face-shaking state, and the eye-moving state, the resultnotification unit 1108 changes the color of the first frame 111 csurrounding the face region to perform the first notification indicatingto a surveillant that the person included in the face region is showingsuspicious behavior. To notify a state of a face region, the appearancesuch as a transparency level, a line type, a thickness, or the like ofthe first frame 111 c may be changed instead of or in addition to achange of the color of the first frame 111 c.

As illustrated in FIG. 9B, when the accumulated score calculated for aface region by the suspicious person determination unit 1107 is largerthan a predetermined threshold, the result notification unit 1108displays a superimposed second frame 111 d to surround the face regionon the image being displayed and thereby performs a second notificationindicating to a surveillant that a person included in a face region iscontinuously showing suspicious behavior. Thereby, it is possible tonotify the surveillant that the likelihood of being a suspicious personis high.

Furthermore, it is desirable that the result notification unit 1108change the appearance of the second frame 111 d, such as a color, atransparency level, a line type, a thickness, or the like, in accordancewith the accumulated score of a face region. For example, the resultnotification unit 1108 displays the second frame 111 d such that thetransparency level is higher for a smaller accumulated score and thetransparency level is lower for a larger accumulated score. Such aconfiguration can facilitate a surveillant to recognize a face regionwhich has a higher likelihood of being a suspicious person. As athreshold for the accumulated score, values defined in advance throughan experiment or a simulation may be used or values input by the inputdevice 119 may be used.

The method of notification performed by the result notification unit1108 is not limited to the method using the display 111 connected to thesuspicious person detection device 110, and an electronic signboard(digital signage), a robot, a speaker, a lamp, or the like installedwithin a shop may be used. The notification contents may be thosetargeted to the suspicious person without being limited to thosetargeted to the surveillant. For example, the result notification unit1108 may output a message or a voice such as “Shoplifting is a crime”from an electronic signboard, a robot, or a speaker located near asuspicious person determined by the suspicious person determination unit1107. With such a configuration, precautions against wrongdoing of asuspicious person can be automatically taken, and labor such asquestioning or the like by a surveillant can be reduced.

FIG. 10 is a front view of the mobile terminal 130 displaying anotification of a detection result from the result notification unit1108. When the accumulated score calculated for a face region by thesuspicious person determination unit 1107 is larger than a predeterminedthreshold, the result notification unit 1108 transmits informationincluding a detection result for the face region to the mobile terminal130, and the mobile terminal 130 displays a notification screen 130 a onthe display thereof.

The mobile terminal 130 displays a face region 130 b extracted by theface region extraction unit 1103 based on information received from theresult notification unit 1108. Further, the mobile terminal 130 displaysa state indication 130 c based on information received from the resultnotification unit 1108. The state indication 130 c includes capturingtime of a face region, a camera ID used for identifying a camera thathas captured a face region, a state determined by the suspicious persondetermination unit 1107, and a text representing an accumulated score.When a face region corresponds to at least one of the mismatch state,the face-shaking state, and the eye-moving state, the mobile terminal130 displays a text representing the state as a state included in thestate indication 130 c, and when a face region corresponds to none ofthese states, the mobile terminal 130 displays “normal”. Each of thenotification screens 111 a and 130 a illustrated in FIG. 9 and FIG. 10is an example, information to be displayed and the display schemethereof may be changed.

The result notification unit 1108 may select a target for notification,that is, a device to be notified of the notification in accordance withthe accumulated score of a face region. For example, the resultnotification unit 1108 performs notification by using the first mobileterminal 130 when the accumulated score is a first value and performsnotification by using the second mobile terminal 130 in addition to thefirst mobile terminal 130 when the accumulated score is a second valuethat is larger than the first value. In such a configuration, if a shopmanager owns the first mobile terminal 130 and shop clerks own thesecond mobile terminals 130, it is possible to notify only the shopmanager when the accumulated score is small and notify many shop clerkswhen a person with a high accumulated score who is highly likely to be asuspicious person appears. Not only the mobile terminal 130 but also thedisplay 111 may be selected as the target for notification. Further, theposition of a target device for notification may be selected inaccordance with the accumulated score. In this case, the resultnotification unit 1108 performs notification by using the first mobileterminal 130 located near the suspicious person when the accumulatedscore is a first value and performs notification by using the secondmobile terminal 130 located more distant from the suspicious person thanthe first mobile terminal 130 in addition to the first mobile terminal130 when the accumulated score is a second value that is larger than thefirst value.

In the process described above, each comparison scheme between thecalculated values from the suspicious person determination unit 1107 andthe result notification unit 1108 and respective thresholds is anexample, and the comparison scheme may be changed as appropriate inaccordance with the definition of the calculated value and thethreshold. For example, “larger than or equal to a threshold” may bechanged to “larger than a threshold”, or the signs of the calculatedvalue and the threshold may be inverted to apply “smaller than or equalto a threshold” or “smaller than a threshold”.

FIG. 11 is a front view of the display 111 displaying a detection resultsearched for by the result search unit 1109. The result display unit1110 displays a result display screen 111 f on the display 111. Theresult display screen 111 f is a screen that displays the searcheddetection result and accepts input of a search condition from the inputdevice 119.

The result display screen 111 f includes a camera ID field 111 m towhich a camera ID of a search target can be input, a time field 111 n towhich a time range of a search target can be input, and a detectionresult list 111 g that displays the searched detection results. Theresult search unit 1109 searches the detection result storage unit 1101for detection results corresponding to a camera ID input in the cameraID field 111 m and a range of time input in the time field 111 n by theinput device 119. The result display unit 1110 displays detectionresults searched for by the result searching unit 1109 in the detectionresult list 111 g on the display 111. In the detection result list 111g, capturing time, an image, a determined state, an accumulated score,an eye direction, and a face direction associated with each face regionincluded in a detection result are displayed. Without being limited tothe above, other information related to a face region may be displayedin the detection result list 111 g.

When one face region is selected through the input device 119, theresult display unit 1110 displays a selection frame 111 h over theinformation associated with the selected face region. Furthermore, theresult display unit 1110 displays the selected face region in a faceimage filed 111 i, displays the face direction as an arrow in a facedirection field 111 j, and displays the eye direction as an arrow in aneye direction field 111 k.

The result display screen 111 f includes an operating button 111 p usedfor play operation and a play time field 111 q in which time to play canbe input. The play time field 111 q may be input with capturing time ofa selected face region or any time input from the input device 119. Theoperating button 111 p includes a play button, a pause button, and astop button, for example. Once the play button is pressed, the resultsearch unit 1109 continuously displays and thereby plays the face regionon the face image filed 111 i from the time of input to the play timefield 111 q. When the pause button is pressed, the result search unit1109 stops playing the face region at the time during playing, and whenthe stop button is pressed, the result search unit 1109 sets the timeback to the play start time and stops playing the face region.

At the same time as play of a face region, an arrow indicating a facedirection is continuously displayed in the face direction field 111 j,and an arrow indicating an eye direction is continuously displayed inthe eye direction field 111 k in accordance with a face region beingplayed. Instead of the arrow indicating a face direction and an eyedirection, points indicating a face direction and an eye direction maybe displayed. In this case, the points indicating a plurality of facedirections and a plurality of eye directions are displayed in a temporalmanner so that the transparency level is lower for closer time to thetime during display and the transparency level is higher for older timefrom the time during display. This enables a surveillant to easilyrecognize transition of the face direction and the eye direction fromthe past to the current.

The result display screen 111 f illustrated in FIG. 11 is an example,and a search condition that can be input, information to be displayed,and a display scheme may be changed.

FIG. 12 is a diagram illustrating a flowchart of a suspicious persondetection method according to the present example embodiment. Thesuspicious person detection method is started when a predeterminedoperation such as press of a button for starting a process on thesuspicious person detection device 110 is detected, for example.

First, the image processing unit 1102 acquires an image that is a movingimage file recorded in the surveillance camera recording device 150 anddivides the image into images that are static image files on a framebasis (step S11). The subsequent process is performed for an image ofeach frame.

The face region extraction unit 1103 extracts a face region including aface of a person by using the method described above from the imageacquired in step S11 (step S12). Further, when there is a face regioncorresponding to the image of the previous frame, the face regionextraction unit 1103 provides the same tracking ID to the extracted faceregion, otherwise, the face region extraction unit 1103 provides a newtracking ID to the extracted face region. When a plurality of faceregions are extracted from an image, the subsequent process is performedon respective face regions to be determined.

The eye direction detection unit 1104 extracts the eye direction byusing the method described above from the face region to be determined(step S13). The face direction detection unit 1105 extracts the facedirection by using the method described above from the face region to bedetermined (step S14). The environment information acquisition unit 1106acquires environment information by using the method described abovefrom the image acquired by step S11 (step S15). Note that steps S13 toS15 may be performed in any order or performed in parallel.

The suspicious person determination unit 1107 uses the eye direction,the face direction, and the environment information acquired in stepsS13 to S15 for the face region to be determined to perform determinationof a suspicious person by using a suspicious person determinationprocess described later by using FIG. 13 (step S100). The determinationof a suspicious person includes the mismatch determination, theface-shaking determination, and the eye-moving determination describedabove. The suspicious person determination unit 1107 determines which ofa mismatch state, a face-shaking state, and an eye-moving state the faceregion to be determined corresponds to or whether or not the face regionto be determined corresponds to none of these states.

If the face region to be determined corresponds to at least one of thestates in step S100 (step S16, YES), the suspicious person determinationunit 1107 adds a predetermined addition score value associated with thestate to the accumulated score of the tracking ID of the face region(step S17). In addition, the result notification unit 1108 uses at leastone of the display 111 and the mobile terminal 130 to perform the firstnotification indicating that a subject included in the face region showssuspicious behavior (step S18). If it is determined that the face regionto be determined corresponds to none of the states in step S100 (stepS16, NO), steps S17 to S19 are not performed.

If the accumulated score is greater than or equal to a predeterminedthreshold (step S19, YES), the result notification unit 1108 uses atleast one of the display 111 and the mobile terminal 130 to perform thesecond notification indicating that the subject included in the faceregion continues to show suspicious behavior (step S20). If theaccumulated score is less than the predetermined threshold (step S19,NO), step S20 is not performed.

If a predetermined operation such as press of a button used forterminating the process is performed on the suspicious person detectiondevice 110 (step S21, YES), the suspicious person detection method ends.If no predetermined operation such as press of a button used forterminating the process is performed on the suspicious person detectiondevice 110 (step S21, NO), the process returns to step S11 and repeatsthe process.

FIG. 13 is a diagram illustrating a flowchart of the suspicious persondetermination process according to the present example embodiment. Thesuspicious person determination unit 1107 determines whether or not theface direction H is directed to at least one object indicated by theenvironment information acquired by step S15 (step S101). Next, thesuspicious person determination unit 1107 calculates the differencebetween the eye direction G and the face direction H (step S102). If theface direction H is oriented to an object and, furthermore, if thedifference between the eye direction G and the face direction H isgreater than a predetermined threshold (step S103, YES), it isdetermined that the face region to be determined is in a mismatch state(step S104). Otherwise (step S103, NO), the suspicious persondetermination unit 1107 does not determine that the face region to bedetermined is in the mismatch state.

The suspicious person determination unit 1107 acquires motion of theface direction H from a predetermined time to the time of capturing theface region to be determined (step S105). Next, the suspicious persondetermination unit 1107 calculates the number of times that certainmotion occurs from a predetermined time to the time of capturing theface region to be determined, and the certain motion is such that themotion of the face direction H is larger than a predetermined thresholdand the center of the motion is oriented to at least one objectindicated by environment information acquired in step S15 (step S106).If the number of times of the motion of the face direction H is greaterthan the predetermined threshold (step S107, YES), it is determined thatthe face region to be determined is in the face-shaking state (stepS108). Otherwise (step S107, NO), the suspicious person determinationunit 1107 does not determine that the face region to be determined is inthe face-shaking state.

The suspicious person determination unit 1107 acquires motion of the eyedirection G from a predetermined time to the time of capturing a faceregion to be determined (step S109). Next, the suspicious persondetermination unit 1107 calculates the number of times that certainmotion occurs from a predetermined time to the time of capturing theface region to be determined, and the certain motion is such that themotion of the eye direction G is larger than a predetermined thresholdin a state where the face direction H is oriented to at least one objectindicated by environment information acquired in step S15 (step S110).If the number of times of motion of the eye direction G is larger thanthe predetermined threshold (step S111, YES), it is determined that theface region to be determined is in the eye-moving state (step S112).Otherwise (step S111, NO), the suspicious person determination unit 1107does not determine that the face region to be determined is in theeye-moving state.

The mismatch determination of steps S101 to S104, the face-shakingdetermination of steps S105 to S108, and the eye-moving determination ofsteps S109 to S112 may be performed in any order or may be performed inparallel.

The processor 112 of the suspicious person detection device 110 is anoperating subject of each step (step) included in the suspicious persondetection method illustrated in FIG. 12 and FIG. 13. That is, theprocessor 112 reads a program used for executing the suspicious persondetection method illustrated in FIG. 12 and FIG. 13 from the memory 113or the storage device 114, executes the program to control each unit ofthe suspicious person detection device 110, and thereby performs thesuspicious person detection method illustrated in FIG. 12 and FIG. 13.

The suspicious person detection system 100 according to the presentexample embodiment performs determination of suspicious behavior basedon environment information indicating arrangement of one or more objectsaround a subject in addition to the face direction and the eye directionof a subject. For example, although a person looking around the left andright of store shelfs in order to search for an intended item whenwalking through an aisle is normal behavior, a use of only the facedirection and the eye direction as with the conventional art may causesuch erroneous detection that determines the person showing suchbehavior as a suspicious person. In contrast, in the present exampleembodiment, since determination is performed by using environmentinformation indicating arrangement of one or more objects such as astore shelf other than a person, erroneous detection as seen in theconventional art can be reduced, and accurate determination of asuspicious person can be performed. Furthermore, the suspicious persondetection system 100 according to the present example embodimentcontrols notification in accordance with the accumulated score added foreach suspicious behavior, and therefore can perform notification onlywhen suspicious behavior is continuously performed or can change theform of notification. This enables a surveillant to focus on a personwho is highly likely to be a suspicious person.

Second Example Embodiment

In the present example embodiment, in addition to an eye direction, aface direction, and environment information, motion informationindicating the presence or absence of motion of a subject is used toperform determination of a suspicious person and notification. Theconfiguration of the suspicious person detection system 100 according tothe present example embodiment is the same as that of the first exampleembodiment, and a method of determination of a suspicious person andnotification is different.

FIG. 14A and FIG. 14B are schematic diagrams illustrating a motiondirection of the subject A in suspicious person detection. Definitionsused when viewing FIG. 14A and FIG. 14B are the same as those of FIG. 3Aand FIG. 3B. When the subject A is moving in a moving direction J asillustrated in FIG. 14A, it can be recognized that the subject A issimply moving to the store shelf C on which an intended item is placed.On the other hand, when the subject A does not move as illustrated inFIG. 14B, it can be recognized that the subject A has already reachedthe store shelf C on which an intended item is placed. In such a state,an action of orienting the line of sight to the aisle direction D whileorienting the face to the item direction E or an action of significantlyand frequently moving the face or the line of sight is considered assuspicious behavior.

FIG. 15 is a block diagram of the suspicious person detection device 110according to the present example embodiment. The suspicious persondetection device 110 according to the present example embodiment has amotion information acquisition unit 1111 in addition to the componentsof FIG. 7. The motion information acquisition unit 1111 acquiresinformation as to whether or not a subject included in a face regionextracted by the face region extraction unit 1103 is moving.Specifically, the motion information acquisition unit 1111 calculateswhether or not the position of the face region extracted by the faceregion extraction unit 1103 has changed from the position of thecorresponding face region in the previous frame. If the position haschanged, the motion information acquisition unit 1111 determines thatthe subject included in the face region is moving, otherwise, the motioninformation acquisition unit 1111 determines that the subject includedin the face region is not moving.

In the present example embodiment, the suspicious person determinationunit 1107 further uses motion information in the mismatch determination,the face-shaking determination, and the eye-moving determination.Specifically, it is determined that a face region to be determined is inthe mismatch state when the difference between the eye direction G andthe face direction H is greater than a predetermined threshold, when theface direction H is oriented to an object other than a person, and,furthermore, when no motion is determined by the motion informationacquisition unit 111. Further, the suspicious person determination unit1107 determines that the face region to be determined is in theface-shaking state of moving the face direction H significantly andfrequently to satisfy a predetermined criterion, when the center ofmotion of the face direction H is oriented to an object other than aperson, and, furthermore, when no motion is determined by the motioninformation acquisition unit 1111. Further, the suspicious persondetermination unit 1107 determines that the face region to be determinedis in the eye-moving state of moving the eye direction G significantlyand frequently to satisfy a predetermined criterion, when the facedirection H is oriented to an object other than a person, andfurthermore when no motion is determined by the motion informationacquisition unit 1111.

The use of the motion information in determination of a suspiciousperson as discussed above can reduce a situation that the subjecthappens to face the direction of the store shelf or the like and this iserroneously detected as suspicious behavior, and more accuratedetermination of a suspicious person can be performed.

As another method, the result notification unit 1108 may select whetheror not to perform notification based on the motion information. In thiscase, the result notification unit 1108 performs the first notificationand the second notification described above by using the display 111 orthe mobile terminal 130 only when it is determined by the motioninformation acquisition unit 1111 that the subject included in the faceregion is moving. By performing notification about a suspicious personin motion as discussed above, it is possible to obtain an advantage ofeasily preventing wrongdoing by performing questioning during thesuspicious person being moving. In contrast, notification may beperformed for only the suspicious person who is not moving.

Other Example Embodiments

FIG. 16 is a general configuration diagram of the suspicious persondetection device 110 according to each example embodiment describedabove. FIG. 16 illustrates a configuration example used for implementingthe function by which the suspicious person detection device 110 detectsa suspicious person from an image. The suspicious person detectiondevice 110 has the eye direction detection unit 1104 that detects an eyedirection of a subject, the face direction detection unit 1105 thatdetects a face direction of the subject, the environment informationacquisition unit 1106 that acquires environment information indicatingarrangement of an object around the subject, and the suspicious persondetermination unit 1107 that determines whether or not the subject isshowing suspicious behavior based on the face direction, the eyedirection, and the environment information.

The present invention is not limited to the example embodimentsdescribed above and may be changed as appropriate within a scope notdeparting from the spirit of the present invention, and theconfigurations described in the example embodiments may be combined asappropriate.

A processing method that stores a program that operates theconfiguration of the example embodiment to implement the function ofeach example embodiment described above (more specifically, a programthat causes a computer to perform the method illustrated in FIG. 12 andFIG. 13) in a storage medium, reads the program recorded in the storagemedium as a code, and executes it in a computer is included in the scopeof each example embodiment. That is, a computer readable storage mediumis also included in the scope of each example embodiment. Further, notonly the storage medium in which the program described above is storedbut also the program itself is included in each example embodiment.

As the storage medium, for example, a floppy (registered trademark)disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, amagnetic tape, a non-volatile memory card, or a ROM may be used.Further, without being limited to one executing a process with only theprogram stored in the storage medium, those operating on an OS toexecute a process in corporation with other software or the function ofan extension board is included in the scope of each example embodiment.

The whole or part of the example embodiments disclosed above can bedescribed as, but not limited to, the following supplementary notes.

(Supplementary Note 1)

A suspicious person detection device comprising:

an eye direction detection unit that detects an eye direction of asubject;

a face direction detection unit that detects a face direction of thesubject;

an environment information acquisition unit that acquires environmentinformation indicating arrangement of an object around the subject; and

a determination unit that, based on the face direction, the eyedirection, and the environment information, determines whether or notthe subject is showing suspicious behavior.

(Supplementary Note 2)

The suspicious person detection device according to supplementary note1, wherein the environment information indicates a position of theobject.

(Supplementary Note 3)

The suspicious person detection device according to supplementary note2, wherein the determination unit determines whether or not the facedirection is oriented to the position of the object and determines thatthe subject is showing the suspicious behavior when the face directionis oriented to the position of the object and a difference between theface direction and the eye direction is greater than a predeterminedthreshold.

(Supplementary Note 4)

The suspicious person detection device according to any one ofsupplementary notes 1 to 3 further comprising a notification unit thatperforms notification based on the determination,

wherein the determination unit accumulates a score associated with thedetermination, and

wherein the notification unit performs the notification based on thescore.

(Supplementary Note 5)

The suspicious person detection device according to supplementary note4, wherein the notification unit performs a first notification based onthe determination and performs a second notification based on the score.

(Supplementary Note 6)

The suspicious person detection device according to supplementary note 4or 5, wherein the notification unit changes a target to be notified ofthe notification based on the score.

(Supplementary Note 7)

The suspicious person detection device according to any one ofsupplementary notes 1 to 3 further comprising a notification unit thatperforms notification based on the determination.

(Supplementary Note 8)

The suspicious person detection device according to any one ofsupplementary notes 4 to 7, wherein the notification unit performs thenotification by using at least one of a mobile terminal thatcommunicates with the suspicious person detection device and a displayconnected to the suspicious person detection device.

(Supplementary Note 9)

The suspicious person detection device according to any one ofsupplementary notes 1 to 8 further comprising a motion informationacquisition unit that acquires motion information indicating presence orabsence of motion of the subject,

wherein, based on the face direction, the eye direction, the environmentinformation, and the motion information, the determination unitdetermines whether or not the subject is showing the suspiciousbehavior.

(Supplementary Note 10)

The suspicious person detection device according to supplementary note 9further comprising a notification unit that performs notification basedon the determination and the motion information.

(Supplementary Note 11)

A suspicious person detection method comprising steps of:

detecting an eye direction of a subject;

detecting a face direction of the subject;

acquiring environment information indicating arrangement of an objectaround the subject; and

based on the face direction, the eye direction, and the environmentinformation, determining whether or not the subject is showingsuspicious behavior.

(Supplementary Note 12)

A program that causes a computer to execute steps of:

detecting an eye direction of a subject;

detecting a face direction of the subject;

acquiring environment information indicating arrangement of an objectaround the subject; and

based on the face direction, the eye direction, and the environmentinformation, determining whether or not the subject is showingsuspicious behavior.

The invention claimed is:
 1. A suspicious person detection devicecomprising: a memory configured to store instructions; and a processorconfigured to execute the instructions to: detect an eye direction of asubject; detect a face direction of the subject; determine that thesubject is showing suspicious behavior when the face direction isoriented to a position of an object around the subject, and a differencebetween the face direction and the eye direction is greater than apredetermined threshold; perform notification based on thedetermination; accumulate a score associated with the determination;perform the notification based on the score; and perform a firstnotification based on the determination and perform a secondnotification based on the score.
 2. The suspicious person detectiondevice according to claim 1, wherein the processor is further configuredto execute the instructions to acquire environment informationindicating the position of the object.
 3. The suspicious persondetection device according to claim 1, wherein the processor is furtherconfigured to execute the instructions to change a target to be notifiedof the notification based on the score.
 4. The suspicious persondetection device according to claim 1, wherein the processor is furtherconfigured to execute the instructions to perform the notification byusing at least one of a mobile terminal that communicates with thesuspicious person detection device and a display connected to thesuspicious person detection device.
 5. The suspicious person detectiondevice according to claim 1, wherein the processor is further configuredto execute the instructions to: acquire motion information indicatingpresence or absence of motion of the subject; and determine, based onthe face direction, the eye direction, environment information, and themotion information, whether or not the subject is showing the suspiciousbehavior.
 6. The suspicious person detection device according to claim5, wherein the processor is further configured to execute theinstructions to perform notification based on the determination and themotion information.
 7. A suspicious person detection method performed bya processor in a computer, the method comprising: detecting an eyedirection of a subject; detecting a face direction of the subject;determining that the subject is showing suspicious behavior bydetermining that the face direction is oriented to a position of anobject around the subject, and a difference between the face directionand the eye direction is greater than a predetermined threshold;performing notification based on the determination; accumulating a scoreassociated with the determination; performing the notification based onthe score; and performing a first notification based on thedetermination and perform a second notification based on the score.
 8. Asuspicious person detection device comprising: a memory configured tostore instructions; and a processor configured to execute theinstructions to: detect an eye direction of a subject; detect a facedirection of the subject; based on the face direction, and the eyedirection determine whether or not the subject is showing suspiciousbehavior when the face direction is oriented to a position of an objectaround the subject; perform notification based on the determination;accumulate a score associated with the determination; perform thenotification based on the score; and perform a first notification basedon the determination and perform a second notification based on thescore.
 9. The suspicious person detection device according to claim 8,wherein the processor is further configured to execute the instructionsto: acquire environment information indicating the position of theobject; and based on the environment information, determine whether ornot the face direction is oriented to the position of the object aroundthe subject.
 10. The suspicious person detection device according toclaim 1, wherein the processor is further configured to execute theinstructions to change a target to be notified of the notification basedon the score.